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ABSTRACT 



This paper provides local educators and policy makers with 
information about educational indicators and their use. It opens by briefly 
tracing the history of the educational indicators movement in the United 
States and then addresses four critical issues or questions about indicators 
and their use: What is an indicator and an indicator system? How can 
indicators and indicator systems be used? How do you identify indicators and 
decide which will be included in an indicator system? and Who needs to be 
involved? The report examines numerous ways in which indicators can be used: 
(1) describe problems quickly and clearly; (2) help in monitoring and 
accountability; (3) assist with evaluations; (4) inform system improvement; 
and (5) advance policy agendas. The article discusses which indicators should 
be included and describes how to use models to select the appropriate 
indicator systems. Also discussed are ways to use frameworks to construct 
indicator systems, tips on deciding how many indicators to employ in a 
system, suggestions for developing criteria for judging indicators, advice on 
using indicator systems' predictive and descriptive capabilities, and 
information on choosing how to use an indicator system. The paper closes with 
a discussion of the implications associated with the use of an indicator 
system. (Contains 55 references.) (RJM) 
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Introduction 

For more than a decade the United States has been engaged in a prolonged campaign of 
educational reform. Federal, state and local policy makers have used a variety of strategies to 
revamp instruction, enhance the professionalism of teachers and restructure the relationship 
between schools and their clients (Elmore, 1990). One reform strategy that has attracted the 
attention of many prominent policy makers and scholars uses “educational indicators” to inform 
the improvement of educational systems. Evidence of the use of indicators is widespread. 
Hardly a month passes without the media trumpeting a comparison of student achievement 
between nations, states, school districts or schools. While many comparisons are based on a 
single standardized test rather than on multiple or composite indicators, they nonetheless 
demonstrate the prominence and power of indicator use. 

Against this national and international backdrop, local educators are confronting the use 
of educational indicators as the basis for accountability and school improvement. For example, 
California’s newly enacted Standards-Based Accountability System will require districts to use 
multiple classroom-based and standardized measures to determine the academic performance of 
students and thus identify sub-standard schools, which will engage in school improvement 
programs. While the actual term “indicators” is not used to describe this system, it is clear that 
indicators are central to this important policy initiative. 

The purpose of this paper is to provide local educators and policy makers with 
information about educational indicators and their use. Although the literature emphasizes 
international, national and state-level use of indicators, here we consider the implications of 
these discussions for the use of indicators by local districts and schools. We begin by briefly 
tracing the history of the educational indicators movement in the United States. We then address 
four critical issues or questions about indicators and their use: What is an indicator and an 
indicator system? How can indicators and indicator systems be used? How do you identify 
indicators and decide which will be included in an indicator system? Who needs to be involved? 
To answer these questions, we collected and analyzed just over one-hundred documents that 
comprise the literature on educational indicators. Our goal is to frame discussions leading to the 
development of CERC's own indicators system. 

A Brief History of Educational Indicators 

Burstein, Oakes and Guiton (1992) observe that historically the U.S. has developed social 
indicators when the nation found itself “groping for a handle on problems still only vaguely 
perceived and at a time of questioning of basic values” (p. 410). This characterizes well the 
emergence of educational indicators in the American educational reform lexicon. 
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The history of educational indicators began in 1867 with the establishment of the U.S. 
Department of Education, which was formed for the expressed purpose of collecting and 
reporting educational statistics. Recent interest in the use of educational indicators as policy 
tools is traceable to the development and use of social and economic indicators beginning in the 
1960's. In the mid-1960's policy makers used economic indicators in recommending tax cuts 
which successfully stimulated the nation’s economy (Burstein, Oakes & Guiton, 1992; 
Shavelson, McDonnell & Oakes, 1991). This success precipitated the development and use of 
social indicators to evaluate the many social programs operating during that period and to justify 
government spending on these programs. 

However, enthusiasm for social indicators soon waned. Policy makers and the general 
public grew disenchanted with the perceived failure of indicators to prescribe effective 
interventions for widespread and growing social problems (Burstein, Oakes & Guiton, 1992). 
The situation was made worse by an overemphasis on the concerns of social scientists at the 
expense of the needs of policy makers. 

Despite these earlier disappointments, in the 1980's educational indicators became 
prominent fixtures on the educational reform landscape. The literature traces the emergence of 
educational indicators to the 1983 publication of A Nation at Risk and other reports, which used 
statistics to document the failings of America’s public education system (Burstein, Oakes & 
Guiton, 1992; Bryk & Hermanson, 1993; Selden, 1994). This was followed in 1984 by the 
Secretary of Education’s first “Wall Chart”, which provided state-by-state comparisons on SAT 
scores, per pupil expenditures and the like. Although many analysts questioned the quality of 
data reported by the Wall Chart, its publication galvanized public attention (Selden, 1994). 

Indicators dramatized the problems of public education by providing seemingly tangible 
evidence. Moreover, critics linked the problems of education to losses in the nation’s economic 
productivity (Decker, Rice & Moore, 1997; Guthrie, 1990; Task Force on Education for 
Economic Growth, 1983). Increasingly, indicators came to be viewed as more than mere 
descriptions of the problem; they provided two tools of reform. First, indicators could serve as 
the basis for holding educational systems accountable (Blank, 1993; Meyer, 1994). Second, 
indicators could be used to uncover the curricular and instructional causes (McDonnell, et al., 
1990) of public education’s failings and thereby provide policy makers with solutions (Odden, 
1990), echoing the 1960's expectations for the potency of social indicators. 

Committed to the potential of indicators as tools of educational reform, organizations and 
agencies at national and state levels coalesced around the development and use of educational 
indicators to improve public education (Bryk &. Hermanson, 1993; Burstein, 1988; Burstein, et 
al., 1992; Selden, 1994). Nationally, in 1984 the Council of Chief State School Officers 
(CCSSO) published a position paper supporting comparisons of states’ academic achievement. 
CCSSO also committed to working with federal and state agencies to enhance their ability to 
collect, analyze and report assessment information. The Council established an educational 
assessment center whose mission was to gather and disseminate information on state assessment 
practices, align state assessment programs and coordinate state, national and international 
assessments of educational quality. 
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1985 was a banner year for the federal government’s involvement with educational 
indicators. The National Research Council published a report recommending that the federal 
government reorganize its efforts to collect and report education data (Burstein, 1988). The U.S. 
Department of Education’s Center for Statistics established plans to reorganize the collection of 
elementary and secondary school data. The Center also sponsored a study to develop a national 
indicators system which used data collected by state assessment programs and supported 
CCSSO’s efforts to develop state-level indicators. In addition, the National Science Foundation 
sponsored the National Academy of Science’s study of mathematics and science indicators (Bryk 
& Hermanson, 1993). 

In 1987 the U.S. government supported the Office for Economic Cooperation and 
Development’s cross-national indicator project, which strengthened efforts to establish a system 
for international comparisons. In the national arena, in 1988 the Hawkins-Stafford Act expanded 
the National Assessment for Education Progress to include the collection of state-by-state data. 
This enhanced coordination of data collection across states. The Hawkins-Stafford Act also 
established the Special Study Panel on Education Indicators to advise the national Center for 
Education Statistics (Bryk & Hermanson, 1993). 

Selden (1994) reports that most states operate educational indicators programs. These 
programs take three forms: One, some states, including Connecticut and Nevada, operate 
indicators systems, which collect and report data on key elements of educational systems. Two, 
other states, including California, Illinois and Louisiana, issue “report cards” which reflect 
schools’ profiles along a number of important dimensions, such as student achievement and 
demographics and elements of instructional programs. Three, still other states employ 
educational indicators in accountability systems. California is moving in this direction by 
enacting its Standards-Based Accountability System. 

History bears important lessons for local policy makers and educators who intend to use 
educational indicators. First, society expects educational systems either to be held accountable 
for outcomes measured by indicators or to use indicators to improve instructional policies, 
programs and practices. Second, the latter is a highly complex endeavor which, according to the 
failures of social indicators in the 1960's, is easily compromised by the limitations of social and 
behavioral science and a lack of relevance to policy makers. 

What Are Educational Indicators? Defining The Concept 

Just what are educational indicators and indicator systems? Educational indicators and 
indicator systems have been defined with differing emphases. While definitions range from 
simple to complex, the literature is nearly always complementary as the simplest conceptions are 
also found within or are the bedrock of the most complex ones. 

Educational Indicators 



The most basic and elementary definitions contend that indicators describe key aspects of 
schooling (OERI State Accountability Study Group, 1988; Selden, 1985). Here indicators are 
statistics that are expected simply to provide information about significant features of the 
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educational system. For example, indicators often describe how students perform on 
standardized tests, what is being taught, or how much is being spent on education. 

Others note that for a statistic to be an indicator, there must be a standard against which it 
can be judged (Dickson & Lam, 1991). An indicator can be compared to itself over time or 
across different schools, districts, or other entities. The “standard for comparison" criteria 
implies that indicators permit evaluation. Given appropriate comparisons, educators can 
evaluate significant features of schooling with indicator data. Evaluation is typically understood 
as a one time activity, usually in respect to a particular program. Hence, evaluation is a rather 
objective and specific exercise that becomes possible when indicators have standards for 
comparison. 

Many purport that the definitive character of indicators is that they act as “vital signs” 
regarding the “health” of the educational system (Hafher & Buchanan, 1992; Jones & Nielsen, 
1994; Kaagan & Coley, 1989; Levesque, Bradby & Rossi, 1996; Oakes, 1986; Nuttall, 1994; 
Raizen & Jones, 1985). There are two aspects embedded in this conceptualization. First, by 
serving as the “vital signs” or “early-warning systems,” indicators permit the monitoring of 
education. In contrast to evaluation, monitoring is typically understood as an ongoing activity. 
Moreover, monitoring has a more complex function than evaluation given this continuous 
tracking of vital signs. 

Second, by providing an assessment of the “health” of a system, indicators provide a 
basis for making value judgments. As Scheerens (1991, p. 371) explicitly notes, indicators are 
“statistics that allow for value judgments to be made about key aspects of the functioning of 
educational systems.” In this context, value judgments can be understood as overall assessments 
that are rather subjective, for there exist no definitive guideposts to determine the health of a 
system. The use of indicators to support subjective health assessments is more complex than 
using them to monitor and objectively evaluate educational systems. The former is purported to 
empower educators by allowing them to draw larger conclusions fi’om their data. 

Other definitions go a step further by invoking a policy criterion. That is, a statistic can 
be considered an indicator only if it is policy-relevant (Burstein, Oakes & Guiton, 1992; Kaagan 
& Smith, 1985; Shavelson et al., 1987; Shavelson, McDonnell & Oakes, 1989; Special Study 
Panel on Education Indicators, 1991). Odden (1990, p. 24) contends that the policy makers 
“want to know what policy levers they can pull in order to improve student 
performance.” Some authors characterize this as an “instrumental usage” of indicators (see 
Bryk & Hermanson, 1992), because educational improvement is treated as an engineering project 
in which indicator data inform the construction of rational policies. These critics note that such 
instrumental use is potentially dangerous, as we may grow to value what is measured over what 
is difficult to measure but nonetheless valuable. 

In summary, our literature review indicates that there are five critical components in 
defining educational indicators: (1) description, (2) evaluation, (3) monitoring, (4) value 

judgments, (5) policy-relevance. These components lie on a continuum and are “additive” as 
indicators are purported to do the most at the end of the list. As a generic synthesis of the 
available definitions, we provide the following: educational indicators are statistics that describe 
key aspects of schooling which permit the evaluation and monitoring of schools, programs, and 



CERC - UCR 



4 



November 20, 1998 



The Use of Educational Indicators 



students. From these activities general assessments (value judgments) of the health of 
educational systems can be derived and policy-relevant information provided. 

Indicator Systems 

Indicators are not arbitrary, isolated measures. They are typically found in sets and 
comprise what is referred to as an indicator system. Like indicators, indicator systems have 
been defined in a variety of ways. Selden (1994, p. 43) describes them as “statistical reporting 
programs representing key aspects of the education system.” This definition emphasizes 
indicator systems as a means for straightforward data presentation. However, the more common 
notion of indicator systems is that they represent relationships between distinct components, 
providing information on the critical, analytical links (Burstein, Oakes & Guiton, 1992; Jones & 
Nielsen, 1994; Oakes, 1986; Shavelson et al., 1987; Shavelson, McDonnell & Oakes, 1989). In 
this approach, indicator systems are based upon a conceptual model of schooling. Indicators are 
developed for each component of the model and the indicator system permits the examination of 
relationships between various components. These relations are often assumed to be causal (e.g., 
increased student expenditures leads to increased student achievement). The implication of this 
notion of indicator systems is that education processes are understood, measurable, and 
representable through data. 

Others focus on the ways in which indicator systems enable educators to monitor the 
statistics on an ongoing basis (Levesque, Bradby, & Rossi, 1996). Given their stability, indicator 
systems support continuous program monitoring. Evaluation, on the other hand, tends to be a 
one time activity. Finally, there are conceptions of indicator systems which highlight policy 
uses. Kaagan and Coley (1989, p. 7) propose that an indicator system is “a fi-amework into 
which an array of indicators are placed for review and analysis, leading to necessary 
modifications of policy and practice.” 

As in the case of definitions of individual indicators, the literature's treatment 
of indicator systems can be seen as a continuum of simple to complex claims of the abilities of 
such systems. The four functions of indicator systems highlighted by these definitions are: (1) 
presentation, (2) representation, (3) monitoring, (4) policy usage. As a whole, indicator systems 
can be considered as a representation of educational systems which enables data presentation, the 
monitoring of key components, and recommendations for policy modifications. Educational 
indicators and indicator systems are of coxuse intertwined as the former comprise the latter. The 
purposes for which indicators are adopted will largely determine the shape of the indicator 
system. For instance, if indicators are used simply to describe some key aspects of schooling, 
then the indicator system may only be a tool for reporting data. On the other hand, if indicators 
are adopted for their policy-relevance, then the indicator system will more likely represent the 
entire educational enterprise and be a monitoring tool that tracks the critical components of the 
educational system. 

How Can Indicators and Indicator Systems Be Used? 

It is apparent, as Selden (1991) observed, that defining educational indicators and 
indicator systems lies principally in determining how they are used. Thus, we turn our attention 
to answering the question: How can indicators and indicator systems be used? 
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In general, the literature identifies five general ways in which indicators can be used by 
educational systems: to describe the state of the system and thus inform public discourse, or 
dialogue; to serve as the basis for accountability; to evaluate policies and reforms; to serve as 
information management systems; and to advance policy agendas. As Porter (1988) observes, 
indicators are about control. They provide policy makers with tools for shaping the structure and 
operation of educational systems. Different uses of indicators offer different forms of control 
over educational systems. 

Describing Systems and Informing Public Dialogue 

The most straightforward use of indicators is descriptive. Authors agree that indicators 
can be used to characterize the state (Nuttall, 1994), status (Burstein, Oakes & Guiton, 1992; 
Edmond, 1992), performance (Burstein, Oakes & Guiton, 1992) and general health (Oakes, 
1986) of educational systems. Others push the analytic function of indicators further by claiming 
that, over time, indicators can chart trends (Guthrie, 1993) or track changes (Oakes, 1986). 
Several authors express strong reservations about moving beyond using indicators to describe the 
status of education or trends over time (Shavelson, McDonnell & Oaks, 1989; 1991). 

Many authors share the view that policy makers can use indicators to describe and state 
problems more quickly and clearly (Oakes, 1989; 1991). This clearly builds on the descriptive 
use of indicators. Shavelson, McDonnell and Oakes (1991) note the limited capacity of 
indicators to inform policy discussions, warning that indicators cannot be expected to replace the 
political process for setting goals and priorities. However, Bryk and Hermanson (1993) advance 
“an enlightenment model” for the use of indicators, arguing that their ultimate fimction will be to 
facilitate broad, well informed and sustained public discourse about the means and ends of 
education. 

Most authors go beyond the claim that indicator systems can reveal existing problems 
and issues. They argue that indicators can serve as an “early warning system” (Nuttall, 1994) 
which can predict future problems (Guthrie, 1993; Oakes, 1986; Shavelson, McDonnell & 
Oakes, 1989). This, of course, exceeds the claims of simple description because prediction 
requires the analysis and interpretation of trends over time in order to anticipate changes in 
educational conditions and outcomes (Smith, 1988). 

Neither describing nor predicting problems and issues, per se, constitute a form of control 
in educational systems. Public discourse, as Bryk and Hermanson (1993) propose, could lead to 
the development of common beliefs and values, which ultimately could contribute to a normative 
form of control over members of an educational system. Despite expressing concerns about 
using indicators to do more than describe and inform, most authors discuss explain how the 
descriptive capacity of indicator systems can contribute to control systems that hold educational 
systems accountable, evaluate programs and reforms or manage information. 

Monitoring and Accountability 

The literature explains that indicators of educational quality can be employed to hold 
educational systems accountable ( Burstein, Oakes & Guiton, 1992; David, 1987; Jones & 
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Nielson, 1992; Nuttall, 1994; Oakes, 1989; Ruby, 1994; Selden, 1994). Several states already 
employ indicators in this way (Nash, 1994; Sanders & Horn, 1994). Richards (1988) identifies 
many ways in which indicators can be used to monitor and thus hold educational systems 
accountable. Two that are particularly relevant to this paper are regulatory compliance and 
performance monitoring. 

Regulatory compliance. Indicators can be used to insure that schools meet determined 
standards. This use of indicators principally focuses on the provision of inputs, such as quality 
teachers, facilities and instructional materials. Regulatory compliance aims to establish 
uniformly adequate instructional services (Richards, 1988). 

Performance monitoring. Indicators can also be employed as the bases for performance 
monitoring (Burstein, Oakes & Guiton, 1992; David, 1987; Nuttall, 1994; Selden, 1994). Policy 
makers establish minimum standards for the student outcomes and hold districts and schools 
accountable for attaining them. Richards (1988) explains that proponents of this use of indicators 
argue that it forces public schools into a market-like situation which will result in more effective 
and efficient operations. That is, schools are sanctioned, positively or negatively, depending on 
whether or not they meet performance standards. 

The use of indicators to hold educational systems accountable for either inputs or 
outcomes employs a regulatory form of control. A central authority, typically the state, uses 
indicators to determine whether or not local schools and/or districts meet minimum criteria and 
rewards those that have met the standard and/or punishes those that have not (Bryk & 
Hermanson, 1993; Burstein, Oakes & Guiton, 1992; Sanders & Horn, 1994; Selden, 1994. 

The literature identifies two limitations of using in accountability systems. First, it 
establishes a minimum requirement to be reached rather than a platfoim from which excellence 
can be launched (Richards, 1988). Second, by focusing on a limited number of outcomes, 
performance monitoring invites educational systems to corrupt indicators (Oakes, 1989), usually 
by providing instruction expressly intended to enable students to meet the criteria, or “teaching to 
the test”. 

Evaluation 



The literature suggests that a major use of indicators is in the evaluation of policies 
(Nuttall, 1994) and reforms (Burstein, Oakes & Guiton, 1992; Porter, 1991; Ruby, 1994). 
Evaluation can, according to some authors, enhance the rational bases of policy analysis by 
providing feedback on the policy effectiveness and efficiency (Nuttall, 1994). Others, however, 
are less sanguine. Shavelson, McDonnell & Oakes (1989; 1991), for example, caution that 
indicator systems cannot evaluate programs, because they lack the rigor of design and depth of 
data and analysis necessary to provide a valid assessment of the relative impact of a program. 

According to Scheerens (1994), control is impossible without evaluation. For, evaluation 
provides feedback which enables education officials to adjust or eliminate existing policies and 
operations and develop new policies in an effort to enhance an educational system’s 
effectiveness. Control based in evaluation is instrumental because it results in the use of 
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effective policies to enhance the system’s outcomes. It employs a technical mechanism which is 
rooted in an understanding of what means, or policies and procedures, produce particular ends. 

Information Management System 

The most comprehensive use of indicators is to inform system improvement, which 
entails all the uses of indicators. In addition to describing and predicting problems and issues 
and evaluating policies and reforms, an indicator system would provide diagnoses and prescribe 
treatments for emergent problems. Indicator systems would become “information management 
systems” (Scheerens, 1991), which continuously gauge outcomes, explain why output goals have 
not been reached (Porter, 1991), and identify interventions that will improve student performance 
(Shavelson, McDonnell & Oakes, 1989). For such information management systems to effect 
student outcomes, some authors argue that they must be employed at all levels of educational 
systems, including policy makers, school-site administrators and classroom teachers (David, 
1988; Jones & Nielson, 1994; Levesque, Bradby & Rossi, 1996). 

In spite of the enthusiasm for using indicator systems as tools for managing systemic 
improvement, many authors question its feasibility. Bryk and Hermanson (1993) offer perhaps 
the most pointed criticism of this position, arguing that it is based on a simplistic view of 
schools, fails to specify how information will influence the educational system, and ignores the 
limitations of research methods in the social and behavioral sciences. Others add that efforts to 
use indicators to manage school improvement will be hampered by the absence of a clear model 
of schooling processes (Oakes, 1986) and the difficulty of measuring important educational 
factors (Burstein, Oakes & Guiton, 1992; Porter, 1991). However, Scheerens (1991) argues that 
the “rational” approach, which requires a clear understanding of cause-effect relations, should not 
be abandoned because individuals and indicator systems do not meet all of the formal 
requirements of technical rationality. 

Advancine Policy Agendas 

The literature identifies one use of educational indicators that otherwise receives little 
attention: advancing policy agendas. This occurs on two levels. First, as Burstein, Oakes and 
Guiton (1992) observe, indicator systems are political entities in that they tend to reflect the 
assumptions of the policy makers who create and employ them. Indicators signal what is 
important and bears attention (Nuttall, 1994). Therefore, policy makers who shape the 
development of indicators systems can thereby shape the educational policy agenda. 

Second, Ruby (1994) adds that policy makers can employ indicators to promote and 
defend their ideological commitments. This is reinforced by David’s (1988) observation that 
policy makers are swayed more by their opinions and beliefs than by data. Thus, in addition to 
utilizing information generated by indicator systems to inform policy making, policy makers can 
employ indicators to bolster preordained positions, either by devising indicators that emphasize 
“pet” concerns or by selecting data that support predetermined positions. This use of indicators 
applies a political form control on educational systems by narrowing the range of policy 
considerations. It thus frames political processes that produce policy decisions. 
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Once policy makers and administrators determine how an indicator system will be used 
to control an educational system, they are confronted with the question of what indicators to 
include. It is to that question that we next turn our attention. 

What Indicators Do We Include? Using Models and Frameworks as Guides 

Much of the literature on educational indicators is intended to be practitioner-friendly. 
This work is full of “how-to” suggestions. It is clear from the literature that there are two 
alternatives for determining which indicators to include in an indicator system. In this section 
we identify and discuss the “model approach” and the “framework approach.” The selection of a 
model or framework of the educational system is critical since it is typically the basis of the 
entire indicator system. Therefore, we highlight the similarities and differences between the 
model and framework approaches. Next we review the parsimony versus extensiveness debate. 
There are differing views on how complex indicator systems should be. This discussion 
identifies the advantages and disadvantages (as well as the contradictions) of each argument. We 
close this section with a review of the frequently cited criteria that should be engaged in 
choosing indicators. 

Usin2 Models to Specify Indicators Systems 

Many contend that in order to develop indicators, one must first adopt a model of the 
educational system (Blank & Gruebel, 1993; Cooley, 1992; Hafher & Buchanan, 1992; Jones & 
Nielsen, 1994; McDonnell et al., 1990; McEwen, 1993; Nuttall, 1994; OERI SASG, 1988; 
Shavelson, McDonnell & Oakes, 1989). A model enables educators to identify the key elements 
of education that they want to be measured. Moreover, models include the theoretical 
relationships between the components. Again, this is based on the assumption that educational 
systems are understandable and measurable. The model approach to developing indicators is 
advocated primarily by those who see indicators as policy-relevant. Models provide clues about 
or highlight areas in need of intervention. The importance of the model cannot be overstated 
since the entire indicator system will be derived from this initial model. The model specifies the 
key areas of interest. These components are operationalized into measurable entities (indicators). 
The collection of indicators make up the indicator system. 

The earliest and most simplistic models considered only “inputs” and “outputs” (i.e., 
Selden, 1985). This rather straightforward notion simply contends that how much and what you 
invest into educational systems will effect what comes out. “Processes” are the intermediate 
category that were soon added to complete the production metaphor (i.e., Raizen & Jones, 1985; 
Shavelson et al., 1987). To this day, the RAND Corporation’s inputs-processes-outputs model 
(Oakes, 1986; Shavelson et al., 1987) is the most widely cited in the literature. As evident in 
Figure 1, RAND’s model of the educational system specifies a multitude of theoretical 
relationships. 

The model is clearly based upon a production function as school, curriculum, teaching, 
and instructional quality (the processes) transform the inputs into outputs. Table 1 details 
RAND’s proposed indicator system for mathematics and science education. This table includes 
the model stages, components, and examples of indicators for each domain. The RAND authors 
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(Shavelson, McDonnell, and Oakes 1989) identify between 5 and 29 potential indicators for 
each component. In Table 1 we have selected three illustrative indicators for each domain. 

The dominance of the RAND model is evident given the many authors who have made 
slight modifications to it. Porter (1991) “complexifies” the “processes” by breaking them down 
into organizational characteristics and instructional characteristics and by adding student non- 
academic activities, course specific teacher quality, and course specific resources dimensions. 
Scheerens (1991) adds a context dimension (consisting of consumer demands, school 
environment, and policy measures at higher administrative levels) and “outcomes” which is 
represented by employment earnings. The Canadian Educational Quality Indicators Initiative 
developed a four-dimensional model that adds the responsibilities of the major partners in 
education (school, family, and society). They also extend student outcomes to include three 
types: types: cognitive, affective, and behavioral (McEwen, 1993). The National Center on 
Educational Outcomes has constructed a conceptual model of outcomes that also borrows 
heavily fi-om RAND’s model. They are specifically concerned with resources, educational 
opportunity and process, and the following outcome measures: physical health, responsibility 
and independence, contribution and citizenship, academic and fimctional iteracy, personal and 
social adjustment, and satisfaction (Vanderwood, et al., 1995). 
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Figure 1: RAND’s Comprehensive Model of the Educational System 
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Table 1: RAND’s Proposed Indicator System 



Model Stage 


Component 


Examples of Indicators 


Inputs 


Fiscal and Other Resources 


• Per-Pupil Expenditure 

• Teacher Salaries 

• Resource Adequacy 




Teacher Quality 


© Teacher Descriptors (Race, Gender, etc.) 

• Experience 

• Comfort With Subject Matter 




Student Background 


• Student Descriptors (Race, Gender, etc.) 

• Socioeconomic Status 

• Courses Taken and Grades 


Processes 


School Quality 


• Course Offerings 

• Student Enrollments 

• Dropout Rates 




Curriculum Quality 


• Content - Breadth and Depth 

• Textbook and Materials Use 

• Coverage of Core Topics 




Teaching Quality 


• Teaching Load 

• Teaching Methods 

• Teacher Planning Time 


Outputs 


Instructional Quality 


• Pupil/Teacher Ratio 

• Instructional Time 

• Access to Labs/Computers 


Achievement 


• Math and Science of All Students 

• ... of College-Bound Seniors 

• ... of Prospective Math/Science Majors 




Participation 


• Extracurricular Activities 

• Math/Science Course Taking 

• Adult Participation 




Attitudes, Aspirations 


• Interest and Liking 

• Social Usefulness 

• Intended College Major 



[adapted from Shavelson et al. (1987, p. 36, 60-2)]l 
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While the RAND model is the most widely cited in the literature, no single model of the 
educational system is universally accepted. Some academics are skeptical of “models” since 
they are very mechanistic and usually engage a production metaphor, which skeptics argue is 
inappropriate for education (see Bryk & Hermanson, 1992; Bramley, 1995). The “inputs- 
processes-outputs” production function is seen as a crude economic analogy whose linear 
reasoning cannot possibly capture what actually occurs in educational systems. Moreover, 
models contribute to the “instrumental usage" of indicator data. In purporting to have the 
educational system mapped out, models suggest that indicator data will highlight areas in need of 
intervention and point to the policy levers that need pulling. This contention is questioned given 
the weight placed upon the model of something too complex to model. The RAND authors do 
carefully note that “[t]he relationships depicted...do not constitute a model in either a strict 
predictive or causal sense”(Shavelson et al., 1987, pp. 9-1 1). Nonetheless, the logic of the model 
itself heavily suggests causality through its linear progression. By claiming that “considerable 
correlational research supports the links among components,” Shavelson et al. (1987, p. 11) still 
invite instrumental usage despite their precautions. 

The model approach to developing indicators is mostly pursued by academics. This 
“objective,” social scientific exercise is based on theoretical constructs and is often completely 
divorced from the daily realities of schooling and local administration. Social scientists tend to 
make such models complex so they can perform statistical analyses. Despite claims of the 
purported policy usefulness of indicators and indicator systems, educators may find them largely 
irrelevant. Models are likely to suggest intervention in areas that are out of the control of local 
educators. Moreover, this model approach may be interpreted as externally imposed upon 
schools. Without having participated in its development, educators may feel threatened and have 
little allegiance to such indicator systems. In the last section of the paper we detail some of the 
issues surrounding stakeholders. 

Using Frameworks to Specify Indicators Systems 

Models are not the only basis from which indicator systems can be constructed. Many 
authors have alternatively proposed “frameworks” for indicators. A framework is generally 
understood as simply a structure for organizing educational domains of interest. By constructing 
or adopting a framework, educators are still able to identify the key elements of interest and 
maintain a stable indicator system without committing themselves to any mechanistic model. 
Frameworks do not usually imply causal relations among the various components and are not as 
prone to inviting instrumental usage of the results. Since frameworks are not attempts to build a 
comprehensive map of the educational system, the data derived from them are more likely to 
inform discussion rather than prescribe a remedy. 

The Council of Chief State School Officers developed a framework that includes student 
outcomes, instructional time, curriculum content, school conditions, teacher quality, and 
resources (Blank, 1986). Oakes (1989) is particularly concerned with school context and focuses 
on three “alterable characteristics” or “enabling conditions:” access to knowledge, press for 
achievement, and professional teaching conditions. Cuttance (1991) identifies four domains for 
indicator development: management and organization, ethos and culture, equal opportunity and 
social justice, and teaching and learning. Gray and Wilcox (1994) are concerned with five key 
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areas: school aims, the ethos of the school, curriculum organization and assessment, curriculum 
implementation, and management and administration. 

The Special Study Panel on Education Indicators (SSPEI) (1991) recommends that 
indicators be organized around major issue areas of significant and enduring educational 
importance. Figure 2 provides the six issue areas and main concepts that the panel suggests for 
indicator development. 

The panel considers "learner outcomes" and "quality of educational institutions" as the 
most essential and provides detailed sub-concepts for both. Table 2 summarizes their 
recommendations. There are between three and six sub-concepts identified for each main 
concept. In Table 2 we provide examples of three sub-concepts for each main concept. In this 
approach, indicators would be developed to measure the sub-concepts. 

As evident, there is quite a bit of similarity between the actual indicators of RAND’s 
proposed system and SSPEI’s recommendations. This does raise concern since we have 
distinguished the model approach from the framework approach. Indeed, the Special Study 
Panel on Educational Indicators purposely distances their work from RAND’s: 

An issue-oriented approach appeared essential...most members of the panel have 
serious reservations about the wisdom of relying on...a model focusing on a 
triumvirate of “educational inputs-educational processes-educational outputs.” 

Most panelists view this approach as flawed because it encourages the view that 
the educational system produces “products”... Such a model may seriously 
mislead decisionmakers... (SSPEI 1991, p. 9). 

Yet, is there really much difference between the issue areas ^d the model? The one 
major difference that was raised in our discussion of indicator systems is that models purportedly 
are a representation of the educational system while frameworks are not. However, despite this 
difference, we find the implications of SSPEI’s and RAND’s approach to be nearly identical. 
While models implicitly invite the instrumental usage of indicator data, SSPEI’s framework 
explicitly does so: 

The strength of indicators, in short, is that they focus attention on critical issues. 

This focusing property means that they can become levers for change; indicators, 
by themselves, can become tools of reform because they are such excellent 
devices for public communication (1991, p. 7). 

While RAND relies on a model based on a production function, SSPEI promulgates an 
“education and economic productivity” issue area. The former represents a problematic 
assumption of congruity between two types of systems. The latter represents the narrow 
assumption that one of the primary purposes of schooling is to prepare students for new, 
high-tech jobs. Both oversimplify educational systems. Moreover, RAND’s model and SSPEI’s 
issue areas framework both represent external impositions of academic “expertise” onto local 
educators. 
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Figure 2: SSPEI’s Six Issue Areas and Their Main Concepts 
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Table 2: Two SSPEI Issue Areas 



Issue Areas 


Main Concepts 


Examples of Sub-Concepts 


Learner 

Outcomes 


Core Content 


• English 

• Mathematics 

• Natural Science 




Integrative Reasoning 


• Scientific & Technological Literacy 

• International Understanding 

• Comprehending Pluralism 




Attitudes and Dispositions 


• Tolerance 

• Self-Direction 

• Responsibility 


Quality of 

Educational 

Institutions 


Learning Opportunities 


• Exposure to Subject Matter 

• Nature of Learning Opportunities 

• Assignment of Teachers and Students 




Teachers 


• Individuals Entering the Profession 

• Pre-Service Training 

• Competence in Classroom Setting 




Conditions of Teachers’ Work 


• Basic Classroom Resources 

• Supporting Resources 

• Influence Over Core Matters of Work 




Schools as Places of Purpose and 
Character 


• Clarity of Mission 

• Human Environment 

• Basic Safety & Order 




School Resources 


• Buildings 

• Libraries 

• Labs & Technology 



[adapted from SSPEI (1991, p. 64, 71)] 
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Although SSPEFs framework shares many of the same problems as RAKD’s model, we 
do feel that the framework approach is distinguishable from the model approach. By definition, 
the model approach is too academic and may be threatening to local educators. The expectations 
of the system will largely determine whether a model or a framework should shape the 
construction of the system. Again, it is important to be clear about your agenda at the outset. If 
you want a policy-relevant information management system, then the model approach is the most 
appropriate. Such usage demands information pertaining to the relationships between various 
educational components. If you are only interested in a system that provides a general 
description of educational processes, then the framework approach is more appropriate. Here the 
informational demands are fewer as the data would be engaged in more limited purposes. 

In deciding whether to rely on a model or framework to guide the development of an 
indicator system, educators should begin by asking themselves a critical question: ^\^at are your 
reasons for wanting an indicator system? If external forces such as the state are requiring data 
about schools, then you have a special set of issues to contend with. When the state requires data 
they usually want it for comparative purposes and schools are responsible for collecting data in a 
compatible fashion. The recent move towards accountability systems resonates with these 
concerns. 

On the other hand, if you want data that will be used solely for local efforts, then there is 
much more flexibility and freedom in developing the system. If local educators are concerned 
with improving instruction, then data can be collected and reported in a fashion that is more 
relevant to teachers. The purposes of the indicator system will also determine how complex it is. 

Parsimony versus Extensiveness 

Another issue that must be confronted in developing indicator systems is deciding how 
many indicators to employ. Both parsimonious and extensive systems have advantages and 
disadvantages. It is widely agreed in the literature that there is no one statistic that can capture 
the pulse of education. Unlike financial quarters, there is no “Dow Jones” average for education 
(SSPEI, 1991). Yet how many is enough? Stecher and Koretz (1991) understand the dilemma 
well: 



There is a fundamental tension between simplicity and comprehensiveness 
that is inherent in the definition of indicators. By design, indicators are 
simple statistics, but they are valued as a way to understand diverse, 
complex systems. An immediate challenge in developing indicator 
systems is to balance simplicity and comprehensiveness. A desire for 
completeness and explanatory power argues for increasing the number of 
variables that are included, the number of ways each is measured, and the 
level of detail of observations. However, indicator systems are valuable 
because they are limited, succinct and parsimonious. The purpose of 
indicators is to illuminate key elements of larger phenomena in a simple 
and concise manner, and this purpose precludes measuring 
comprehensively. One cannot achieve both goals; compromise is required 
(p. 58). 
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In the literature, most advocate compromising comprehensiveness. An extensive 
indicator system would be unmanageable (Shavelson, McDonnell & Oakes, 1989) and too 
complex (Blank, 1993), lacking utility (Porter, 1991). If educators are inundated with data there 
will not be any strong messages sent. Moreover, a comprehensive set of indicators would be 
very costly to develop and maintain (Oakes, 1986). While most simply state that indicators must 
be few in number, others provide concrete suggestions. Dickson and Lam (1991) recommend 
between five and eight indicators, the Council of Chief State School Officers suggests twelve 
(Blank, 1993), and Hafher and Buchanan (1992) recommend in the ballpark of twenty or fewer. 
It is evident that there is a lack of consensus around the notion of “few.” 

There are at least a few critics of parsimonious indicator systems. The Special Study 
Panel on Educational Indicators (1991) contend that the search for a limited number of key 
educational indicators is misguided. A limited set of indicators could not do justice to the 
complexity of the educational enterprise and would end up defining the educational agenda 
instead of reflecting it. Likewise, Porter (1988) warns that parsimony is problematic and may 
strengthen the prevalent beliefs that teaching and learning are straightforward. SSPEI does 
recognize that hundreds of indicators would be overwhelming. “Indicators must be 
comprehensive, yet disciplined enough to be manageable”(SSPEI, 1991, p. 8). For the two issue 
areas for which the panel provides details, there are 35 sub-concepts which could be developed 
into indicators. Others note that the actual number of indicators may effect the reaction of 
educators to indicator systems: 

While understanding the desire of busy policy-makers and managers for a 
limited and simple set of indicators and the researchers’ desire for a 
parsimonious one, there are dangers that arise from keeping the set small. 

The greatest danger is that of corruptibility of the behaviour of those 
whose performance is being monitored. The best-known example is 
teaching to the test... (Nuttall, 1994, p. 23). 

While the RAND authors and the SSPEI differ on the parsimony versus extensiveness 
debate, we would not conclude that these differences are characteristic of the model approach 
and the framework approach to developing indicators. The majority in both camps favor 
parsimony. The main thrust of the parsimonious argument seems to concern policy-usefulness. 
After all, if there are too many numbers, it will not be clear which policy levers need to be 
pulled. Intrinsic to the model approach, there is a contradiction between their calls for 
parsimony and policy-usefulness and their basing of indicator systems on theoretical models. 
The “objective,” social scientific exercise of developing models requires extensiveness so that 
statistical procedures can be adequately performed and are not underspecified. Indeed, for the 
ten components of their model, the RAND authors discuss a total of at least 85 indicators! (see 
Shavelson, McDonnell & Oakes, 1989). This is parsimony? The Massachusetts Association of 
School Superintendents developed indicators for their school system based on RAND’s model. 
Their proposed indicator system includes 71 indicators (Hafher & Buchanan, 1992, pp. 84-5). It 
is simply not clear how this many measures can be policy-relevant. If a few “key” indicators are 
selected, then the objectivity of RAND’s model is compromised by a rather subjective process. 
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In order to further discuss some of these contradictions, we shall now review some of the criteria 
from which indicators are developed and evaluated. 

Criteria for Judging Indicators 

Throughout the literature on educational indicators, scholars provide lists of criteria for 
choosing indicators. The single most frequently cited criteria is in regards to the technical 
quality of the data including the issues of reliability and validity (Blank, 1986; 1993; McPherson, 
1993; Meyer, 1994; Mumane, 1987; Nuttall 1994; Porter, 1991; Shavelson, McDoxmell & 
Oakes, 1989; Smith, 1988; Stronach, 1993; Texas Education Agency, 1996). Reliability 
concerns the quality of measurement and suggests that the same data would be obtained each 
time in repeated observations of the same phenomenon. A measure is valid if it accurately 
reflects the concept that it is intended to measure. Reliability and validity are certainly critical to 
indicator development and use. As Koretz (1992) details, the threats to reliability and validity 
that educational indicators are particularly prone to stem from the characteristics of the indicators 
themselves, the indicator systems, and the ways in which they are used. For example, indicators 
currently are often used to support broad inferences, rely on simple and corruptible measures, are 
context-dependent, are often available for only a sparse array of measures, and are narrower that 
the effects of the policies that they are intended to monitor. One problem with these technical 
criteria is that they are inextricably bound with the model or framework from which the 
indicators are derived. Even if the measures in a meaningless framework are reliable and valid, 
they are still meaningless. Reliability and validity alone do not qualify indicators to be taken 
seriously. 

Another criteria frequently found in the literature is that indicators be derived from a 
model of the education system (Blank & Gruebel, 1993; Cooley, 1992; Hafher & Buchanan, 
1992; Jones & Nielsen, 1994; McDonnell et al., 1990; McEwen, 1993; Nuttall, 1994; OERI 
SASG, 1988; Shavelson, McDonnell & Oakes, 1989). Of course, these are the advocates of the 
model approach to indicators. With this criteria we are placed back into the aforementioned 
problems of the difficulty of modeling such complex phenomena and there being no single 
agreed upon model of the education system. Also, a popular criteria for educational indicators is 
that they be policy-relevant and include aspects that can be manipulated by policies or programs 
(Cooley, 1992; David, 1987; Hafiier & Buchanan, 1992; Jones & Nielsen, 1994; Nuttall, 1994; 
OERI SASG, 1988; Shavelson, McDonnell & Oakes, 1989; Smith, 1988). To require that an 
indicator measure something that is alterable is by definition constraining the potentialities of the 
indicator system. This automatically locks you into the “way things are always done,” the status 
quo. To create indicators that are specifically policy-relevant implies that the “levers for change” 
are already known. 

Many also cite feasibility (Blank, 1986; 1993; Hafher & Buchanan, 1992; Nuttall, 1994; 
Raizen & Jones, 1985; Shavelson, McDonnell & Oakes, 1989; Smith, 1988; Stronach, 1993; 
Texas Education Agency, 1996) and cost (Porter, 1991; Fitz-Gibbon, 1996; Hafher & Buchanan, 
1992; Nuttall, 1994; Raizen & Jones, 1985) as important criteria for indicators. It is interesting 
to note that cost is cited as an important criteria much less often than the others. 

Finally, a few are cognizant of some of the problems of the external imposition of 
indicator systems onto local districts and schools. Indicators must produce information that is 
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useful at the local, state, and national levels and indicator systems should respect the autonomy 
of and be sensitive to differing state and local definitions (NEGP, 1993). Nuttall (1994) cites 
criteria from a pluralistic view which contends that all the partners in education need to have a 
sense of ownership of the indicator system. One way to help ensure this is to adopt a grassroots, 
participatory approach to developing indicators. Levesque, Bradby, and Rossi ( 1 996) go a step 
further and claim that indicators should be rooted in local goals. 

What is clear to us in these criteria is the contradictions between some of them. As 
mentioned earlier, the model and policy criteria conflict with each other. The model approach 
ends up being too complex for policy. Also, the feasibility criteria potentially conflicts with the 
model criteria as well. Some parts of the educational system are simply not able to measured and 
adequately quantified. Those who advocate the local criteria are not supportive of the model 
^ approach and it is clear that these criteria would conflict. The former is grassroots and 

participatory while the latter is externally imposed by the scientific community. The technical 
criteria tend to contradict the cost criteria since the steps to ensure reliability and validity are 
typically expensive. We could go on, but the point is that these criteria are not all they are 
purported to be. There is no “cookbook” approach to developing indicators and indicator 
systems. Criteria should be used to guide the process, but there will inevitably be tensions. 

Who Should Be Involved? 

An important issue that receives relatively limited attention in the literature concerns who 
should be involved in developing and operating an indicators system. How an indicator system 
will be used and, thus, the control it will exert over an educational system determines in large 
measure who should be involved in specifying what indicators will be included and who will use 
indicator data to inform decision and policy making. 

Informing Policy Discourse 

The literature indicates that policy makers can draw on the descriptive and even 
predictive capacities of indicators systems to identify issues and trends and thus inform policy 
discourse. Bryk and Hermanson (1993), in particular, emphasize this use of indicators, warning 
that major theoretical and methodological limitations seriously inhibit the use of indicator 
§ systems to manage system improvement. Bryk and Hermanson argue that schools do not possess 

the chief qualities of highly rationalized bureaucracies. They, instead, treat schools and districts 
as communities, which are based on a convergence of values. Thus, Bryk and Hermanson 
explain that individuals at all levels of schools and districts — including parents, teachers, 
administrators, policy makers and, when appropriate, students — should be engaged in the design 
and use of indicators systems. 

Accountability 

As we have suggested, policy makers and educational agencies exert regulatory control 
when an indicator system is used to hold educational systems, fi'om schools to states, accountable 
for the quality of educational inputs and/or the level of outcomes. Thus, accountability systems 
provide control to centralized authorities. This clearly marks who both selects the inputs and/or 
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outcomes for which the educational system will be held accountable and sets the standards for 
what is acceptable: centralized policy makers. This is consistent with the assumption on which 
accountability systems are based: Policy makers exert market-like pressure by monitoring inputs 
and/or outcomes (Richards, 1988). Subordinate systems are left to their own devices to do 
whatever they deem necessary to reach the standards. The central authorities then positively or 
negatively sanction the subordinates systems depending on whether or not they reach established 
standards. 

Evaluating Policies and Reforms 



As we noted above, many authors suggest that indicator systems can be used to evaluate 
the effectiveness of policies and reforms. This use is instrumental because it enables policy 
makers to use the feedback provided by evaluation to improve the outcomes of educational 
systems (Scheerens, 1994). This implies that two groups are involved in selecting indicators and 
using the results of this use of indicator systems. The first are centralized policy makers whose 
decision making will be predicated at least in part on the results of evaluations. The second are 
either outside consultants or administrators within the educational system who conduct the 
evaluation. In either case, the evaluators would guide policy makers in selecting indicators and 
in interpreting the evaluation results. 

System Improvement 

Scheerens (1991) proposes the most complex and arguably sophisticated use of indicators 
systems: providing an information management system to guide system improvement. We 
argued above that this use of offers technical control because it provides “policy levers” (Odden, 
1990) to improve system outcomes. The creation of such a system must be based on a 
theoretical model of schooling which identifies crucial inputs, conditions, processes and 
outcomes. Thus, social scientists would work with central policy makers to design the indicator 
system, analyze indicators data and assist in the interpretation of results. 

Some authors disagree. They contend that in order for indicators to be useful to the 
improvement of system performance, stakeholders at all levels of the system must be involved in 
choosing indicators and in using indicators data. Citing research that indicates that information 
seldom affects decision making and planning, David (1988) claims that teachers and principals 
will be more likely to use indicators to improve instructional practice and programs if they are 
involved in building the indicator system. Levesque, Bradby and Rossi (1996) also note that 
people will not use data to develop improvement strategies if they perceive that they were 
collected for other reasons. They cite an evaluation study conducted by RAND and Management 
Planning Research Associates for the National Center for Research in Vocational Education 
which indicates that local vocational administrators and teachers were more likely to use 
performance measures to improve practice when they participated in developing the measures 
and related instruments. She also argues that evaluations conducted by outside consultants are 
often ignored by local educators and reports that some local educators are thus working to build 
the capacity of local districts and schools to self-evaluate. 

Porter (1998) offers yet another perspective on the question of who should be involved in 
designing and using indicators systems for system improvement. He agrees that district and 



CERC-UCR 



21 



27 



November 20, 1998 



The Use of Educational Indicators 



school level assessments are more likely to influence instructional content and practice in the 
classroom than more distant testing programs. He adds that, while educational indicators cannot 
provide the detail necessary to reveal the nature of individual teachers’ practices, indicators 
“could potentially describe the practices of a representative sample of teachers and other teachers 
who wished to do so could compare their own practices” (Porter, 1988, p. 507). 

Advancing Policy Agendas 

Porter (1988) ultimately concludes, however, that the real potential of indicator systems 
does not lie in their capacity to provide information about the instructional practice of teachers. 
Rather, he suggests that indicator systems can be used as a political tool. Whoever shapes 
indicators systems can use them to fix attention on certain conditions and outcomes of 
educational systems and thus advance particular educational policy agendas. Most discussions 
explain that indicator systems will serve the needs and interests of policy makers (Burstein, 
Oakes & Guiton, 1992; Porter, 1988; Nuttall, 1994). But, Porter argues that other stakeholders 
could advance their interests by influencing the design of indicator systems. Specifically, Porter 
explains that if teachers participated in designing indicator systems they would advance their 
professional status by influencing what will be taught and how. Moreover, if teachers are 
involved in the development of indicator systems, those systems would have greater legitimacy 
with cuid thus influence on all teachers. 

Discussion 

The literature on educational indicators is highly complex and crosscut by ambiguous and 
sometimes conflicting messages. Thus, it provides few, if any, clear-cut answers regarding how 
local school systems might use indicators to improve academic performance. Instead, a careful 
reading of the literature raises several issues with which local policy makers and educators will 
have to wrestle in developing and operating indicator systems. Here, we discuss three. They 
begin with how the indicator system will be used. This basic choice will be tempered by two 
other, related considerations: what are the information demands of the different uses of indicator 
systems and how do these demands affect the feasibility of using indicator systems in these 
ways. 

Choosing How to Use an Indicator System 

The first choice that local school districts will have to make is whether to invest in an 
indicator system in the first place. In many ways, however, the answer is inevitable, for 
expectations regarding the use of indicators to improve the performance of public education in 
the United States run high. The history of educational indicators in the U.S. reminds us that 
policy organizations, such as the Council of Chief State School Officers, and government 
agencies, including the Department of Education, have promoted educational indicators as 
important tools for educational reform. Individual states soon followed by adopting indicator 
systems for a variety of purposes. Just this year, California enacted its Standards-Based 
Accountability System which will hold schools accountable for the academic performance of 
students. Thus, in states such as California, at least, the question of whether local districts will 
use indicator systems has already been answered. 
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Once it has been determined that a local educational system will use indicators, the first 
issue that must be addressed is how the indicators will be used. In states where indicator systems 
already exist, local districts and schools are obviously bound to their state’s uses of indicators, 
which usually take some form of accountability. However, local districts may also choose to use 
indicators in other ways. The literature identifies five: to inform policy discourse, to hold 
subsystems accountable, to evaluate policies and programs, to provide an information 
management system that informs system improvement and to advance a policy agenda. How 
does a local system select its use of an indicator system? As we noted, the answer to that 
question turns in part on the type of control that a district wants to exert. 

Informing policy discourse exerts the most indirect control. It is aimed at building shared 
understanding among educational stakeholders of the issues and problems confronting the 
district and its schools. This use of indicators could result in the development of a normative 
form of control, the sort of control that a community exert over its members through a system of 
shared values. However, it seems unlikely that districts will be satisfied with this form of 
control. Most of the literature advances a more ambitious agenda. It suggests that indicator 
systems can monitor schools and districts to hold them accountable for providing sufficient 
educational inputs and producing adequate academic outcomes, evaluate policy and reform and, 
ultimately, serve as information management systems to guide the improvement of whole 
educational systems. Simply facilitating policy discourse does not seem to satisfy policy 
makers’ or scholars’ desire for greater control over educational systems. 

Thus, local educational agencies may choose to use an indicator system to hold schools 
accountable, just as the state intends to in California. This exerts a regulatory form of control. 
Schools that do not meet minimum standards for inputs, instructional processes and/or academic 
outcomes would be punished and/or those that do meet the standards would be rewarded. 
Proponents of this form of control argue that it leaves local schools fi-ee to determine how best to 
reach standards. 

Using an indicator system to evaluate policies and programs and as an information 
management system both would exert technical control on local educational agencies. 
Evaluating the impact of specific policies or programs provides feedback, which local policy 
makers and administrators could employ to make mid-course adjustments, eliminate policies or 
programs or adopt new ones. 

An information management system would take this several steps further. If proponents 
of this use of educational indicator systems are correct, such a system would provide ongoing 
information that people throughout the local educational agency could use to improve policies, 
programs and practice. To many authors who contributed to the literature on educational 
indicators, this is the ultimate use of indicators for contributing to the improved performance of 
educational systems, ranging from nations to local schools and even classrooms. 

The final use of an educational indicator system is to advance a policy agenda. The 
literature suggests that policy makers can accomplish this by selecting indicators to bring 
attention to particular conditions or outcomes and thus exerts political control. This use of 
indicators differs from the others because it is not aimed at affecting the internal operations of 
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local districts and schools, so much as it focuses on the process of policy making itself. The 
literature pays only limited attention to this use of indicator systems, reflecting its emphasis on 
regulative and technical forms of control. 

Information Demand and System Feasibility 



The choice of how an indicator system will be used has implications for the information 
demands that will be placed on local educational agencies. The level of demand, thus, may 
weigh in the decision on how the indicator system will be used. Information demand has two 
dimensions. First, information is more or less dense, depending on the amount of data that is 
required. Density is function of the number of indicators included in a system and the number of 
data sources that must be tapped. Second, information demand varies in complexity, depending 
on the degree to which an indicator system examines relationships between indicators. 
Information demands vary substantially across the different uses of educational indicator 
systems, ranging from the high density and complexity of an information management system to 
the low density and relative absence of complexity in the descriptive data needed to inform 
public discourse. 

Moreover, the level of information demand, in part, determines the feasibility of 
supporting and operating an indicator system, because information comes at a price. The issue 
of the cannot be taken for granted, for history provides a potential and not hopeful precedent. 
Buoyed by the successful use of economic indicators to enact policy that stimulated the U.S. 
economy, policy makers in the 1960’s urged the development of social indicators. However, 
expectations were quickly dashed when policy makers were unable to use social indicators to 
prescribe interventions to cure the nation’s social problems. 

Using an indicator system to hold schools accountable does not require particularly dense 
or complex information. A general framework, rather than a detailed theoretical model, could be 
employed to identify key conditions and outcomes that would be included in the indicator 
system. Such a system would be more parsimonious than extensive, focusing on a manageable 
set of indicators. In addition, analysis of indicator data would not examine relations between 
indicators but determine whether or not minimum standards had been reached. However, there is 
an important cost associated with using an indicator system for accountability. The literature 
warns that, when used to hold systems accountable, indicators run the distinct risk of being 
“corruptible”; they can be manipulated and thus may not present a valid indication of an 
educational system’s performance. The most common means of corrupting narrow indicators of 
student performance, such as standardized achievement tests, is simply to “teach to the test.” To 
insure the validity of the accountability system, the educational system would have to incur the 
cost of limiting, through oversight, the possibility that teachers and other stakeholders would 
corrupt indicators. Thus, using an indicator system to hold schools accountable appears to be 
economically feasible, and may explain why so many states have opted for this use. The 
downside is that outcome indicators, at least, may not provide valid information about system 
performance. 

The demand for information beconies much more complex if an indicator system is used 
to evaluate programs, policies and/or reforms. In fact, the literature indicates that indicator 
systems may not be particularly well suited to this purpose. Some authors caution that indicator 



CERC - UCR 



24 



ri 



November 20, 1998 



The Use of Educational Indicators 



systems, which reflect the overall “health” or status of educational systems, lack the rigor of 
design and depth of data and analysis necessary to provide valid assessments of the impact of 
policies or programs. A rigorous indicator system that provides in-depth data about key 
conditions and outcomes would require informational density and complexity. A detailed 
theoretical model of schooling would be employed to identify indicators. The number and types 
of indicators, or information density, would likely be greater than in accountability systems, 
because evaluation would require assessing the relative impact of a variety of conditions in 
educational systems in order to estimate the contribution of the focal program or policy. This 
would also require examining relationships between indicators, thus adding information 
complexity. Thus, it is likely that costs will rise if an indicator system is used for purposes of 
evaluation. One alternative is to operate a fairly parsimonious indicator system to assess the 
general status of the system, which would provide baseline data for more extensive evaluation 
^ studies of particular programs and policies. 

Information demand would be even greater if an educational indicator system is to be 
used as an information management system to guide system reform and improvement. The 
information would be complex, for an information management system would most certainly 
have to be based on a detailed and comprehensive model of schooling. Even the most ardent 
promoters of educational indicator systems acknowledge that educational researchers caimot 
agree on such a model. This raises serious reservations about the technical, let alone financial, 
feasibility of using indicator systems for evaluation or as information management systems. 
Moreover, many of the most important inputs, conditions and outcomes of educational systems 
are difficult and, thus, costly to measure. Finally, the literature indicates that teachers and site- 
level administrators would have to be involved in designing and using the system if it is to effect 
improvements in instructional programs and practice. Thus the technical and human costs of 
using indicators in an information management system would be high, perhaps prohibitively so 
for small and/or low wealth schools and districts. 

The information demands of informing public discourse can be relatively limited. If 
basic information regarding the health or status of a local system are required, stakeholders can 
employ a general framework, or set of rubrics, to determine the conditions and outcomes for 
which indicators will be developed and data collected. Descriptive statistics can be calculated 
and, overtime, trends can be charted. Thus, information complexity is low. Such an indicator 
^ system would also be relatively parsimonious, for a more extensive system could well 

overwhelm rather than inform public discussion. Thus, it appears that using an indicator system 
to inform public discourse is quite feasible. But would local policy makers and administrators be 
satisfied to use an indicator system in this way and not seek to use information to exert greater 
control? 

The literature pays only limited attention to using indicator systems to advance policy 
agendas and thus does not directly address the information demands of this use. However, it 
seems that the information demands would be relatively low. Policy makers would establish a 
general fi-amework based on their particular policy interests. The fi-amework would guide their 
choices of the conditions and outcomes for which indicators would be developed. Thus, the 
information demands would be similar to those of using an indicator system to inform public 
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discourse or hold subsystems accountable. The difference would be that the motivation is 
expressly political. 

The information demands and thus the potential cost of some uses of indicator systems 
can be quite high. The highest information demands are associated with the technical control 
exerted through policy and program evaluation and by information management systems. Both 
require high density and complexity of information. While using indicators to provide regulatory 
control through an accountability system has relatively modest information demands — neither 
dense nor complex — the potential of corruptibility brings with it the cost of oversight. Which 
leaves two uses of indicators: facilitating discourse and setting policy agenda. Both have 
relatively modest information demands, requiring neither dense nor complex indicator data and 
analysis and, thus, appear to be feasible uses of indicator systems. However, neither will enable 
policy makers and administrators to control the operations or outcomes of educational systems. 

That cost and feasibility may undermine efforts of local districts and schools to profit 
from the use of indicator systems suggests a possible strategy that may be especially appealing to 
small and low-wealth systems: collaboration. Several districts could join together to design and 
operate an indicator system, thereby sharing costs. Such collaboration offers other advantages. 
If the system is used to evaluate programs or policies, the impact of contextual differences across 
districts on policy effectiveness could be assessed. If indicators are used in an information 
management system, the involvement of multiple districts could provide two kinds of 
comparisons. One, the influence of district conditions on student and school performance could 
be examined. Two, a study could be conducted on differences in information utilization across 
districts. Finally, if the indicator system is used to forge a policy agenda by focusing on 
particular conditions or outcomes, a bloc of districts could potentially have much greater regional 
and even statewide influence that a single district. 
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